1. Technical Field
The present invention relates in general to an improved method of testing and verification, and, in particular, to an improved method of testing and verification in a simulated-hardware environment. Still more particularly, the present invention relates to an improved method for performing testing of a simulated direct access storage device within a simulation environment.
2. Description of the Related Art
The modern trend toward constant and rapid improvement in technology has forced technology companies to adapt to a market environment where the saleable life of a technology is limited and frequently prematurely terminated by the frenetic pace of innovation. In order to maximize the useful life of a product when facing a shrinking window of time between commercialization of an invention and displacement by superior innovations, technology companies have undertaken every possible effort to reduce product cycle time. Manufacturers frequently turn to virtual prototyping and simulation testing techniques to enable them to test products before the design reaches the factory floor. Virtual prototyping and simulation testing techniques carry the added benefit of reducing the cost of development. But, as will be discussed below, current virtual prototyping and simulation testing techniques also add risk and costs to the design process.
Simulation testing techniques inject unique complications into the design and testing of complex technological products. A recurring and somewhat daunting problem relates to the need to test the interaction of separate systems, both of which are in development at the same time, to external stimuli. Where separate systems such as underlying hardware and control software form integral parts of the product delivered to the customer, integration testing must be used to verify that the parts will, in fact, perform together as specified. Particularly in modern data processing and storage systems, where the hardware and software form equally critical parts of the product, it is necessary to rigorously test their interactions.
Testing the interaction of the hardware-under-development with the software-under-development sounds inherently simple; one need only produce a prototype of the hardware and then monitor the performance of the control software on the completed hardware platform. The unfortunate reality is that time-sensitive cycles of parallel development often require integration testing of the software and the underlying hardware prior to production of the hardware itself. Testing must frequently occur before a working prototype of the hardware exists.
In the direct access storage device industry, as in many others, the answer has been to produce a virtual prototype of the relevant hardware. The virtual prototype is a data processing system program, operating in a testing simulation environment, which models the expected behavior of the hardware. In order to perform integration testing, a specially engineered version of the software-under-development is conventionally loaded into the simulation environment and sends commands to the virtual hardware. Such methods have allowed for testing of the expected interactions of non-existent hardware with the software designed to control the virtual hardware, but the use of a specially engineered version of the software carries its own problems.
Current testing methods call for the insertion of special program instructions into the software, which will presumably be removed from the software at a later time. As an example, a testing and verification group might need to determine the combined response of the software and the hardware to the presence of a thermal abnormality in the underlying hardware. To do this, lines of code in the software will be inserted, which will, in effect, write a thermal error code to the underlying hardware. The hardware will then react to this error code, taking some measures based on its own firmware, and passing the error back to the software for further scrutiny.
The first, and most obvious problem with this approach, is that integration of the special error-code instructions into the software-under-development changes the performance dynamics of the software. Machine cycle times are skewed by the need to perform the lines of code that pass error codes to the hardware. This reduces the accuracy of the simulation by causing the development and testing version of the software to exhibit behavior that will not be present in the production version of the software. Because the timing of microprocessor operations can prove critical to successful operation of the completed system, this mere problem of timing due to presence of testing instructions can severely cripple the usefulness of the integration testing. Injecting work-arounds to correct for delays in software operation can introduce an added factor of unreliability. Simply put, the fidelity of the test to the software that will actually form a component of the finished product is substantially undermined.
Another problem relates to the burdensome nature of manual insertion of the testing commands into the software-under-development. The manpower involved in properly coding the test instructions into the software-under-development adds cost to the development process and creates version management issues. Frequently, a separate testing and verification group, composed of persons less familiar with the software-under-development than the people who have written the software, performs the integration testing. The cost of providing the testing and verification team with the requisite familiarity with the underlying software, and then having them perform, and later remove, changes to the software, adds materially to the cost and number of skilled programmers necessary to deliver the product. The fact that the changes to the software necessary to facilitate testing must be made manually also discourages thoroughness in testing by imposing an unacceptably high time-cost for each test performed.
Finally, the presence of human error in the process of testing creates real problems. One such problem is the problem of version management. Once simulation testing has begun, the programmers, who are developing the software, must take care not to make revisions to the software without insuring that the revisions are passed to the testing and verification group. Frequently, in order for the testing and verification group to make the modifications that need to be made in order to test the software, they must isolate the code from further tinkering by the software design team. The frequently unhappy result is that revisions to the software are lost because the revisions are not inserted into the most current version-under-test, or that they are inserted and are never tested because the testing and verification group had no notice of the insertion of the code. Occasionally, testing and verification instructions are inadvertently left in the software product after testing is completed, causing the finished product to display anomalous behavior that was not anticipated by the designers.
All of the above-mentioned problems undermine the effectiveness of current testing techniques, and a remedy that would both lower the cost and increase the effectiveness of simulation testing is desired. What is needed is a separation of the testing function from the development of software through a method of simulation testing that automates the error code process and removes the need to insert testing instructions into the software-under-development.